How Akamai delivers your packets - the insight

1. How Akamai delivers your packets - the insight
Christian Kaufmann
PLNOG 6
March 2011

2. © 2008 Akamai
What is a Content Distribution Network?
• The RFCs and Internet Drafts define a Content
Distribution Network, “CDN”, as:
Content Delivery Network or
Content Distribution Network. A
type of CONTENT NETWORK in which
the CONTENT NETWORK ELEMENTS are
arranged for more effective
delivery of CONTENT to CLIENTS.

3. © 2008 Akamai
What is a Content Distribution Network
- In English?
• A CDN is an overlay network, designed to
deliver content from the optimal location
• In many cases, optimal does not mean
geographically closest
• CDNs are made of distinct, geographically
disparate groups of servers, with each group
able to serve all content on the CDN
• Servers may be separated by type
• E.g. One group may serve Windows Streaming
Media, another group may serve HTTP
• Servers are not typically shared between media
types

4. © 2008 Akamai
What is a Content Distribution Network
- In English?
• Some CDNs are network owned (Level 3,
Limelight, at&t), some are not (Akamai, Mirror
Image, CacheFly, Panther Express)
• Network owned CDNs have all / most of their
servers in their own ASN
• Non-Network CDNs can place servers directly
in other ASNs
• This means things like NetFlow will not be useful for
determining traffic to/from non-network CDNs

5. © 2008 Akamai
The Akamai System
Resulting in traffic of:
5.0 Tbps peak traffic
36 petabytes / day
790+ billion hits / day
436+ million unique clients IPs / day
The Akamai EdgePlatform:
85,000+
Servers
1700+
POPs
72+
Countries
950+
Networks
660+
Cities

6. © 2008 Akamai
How CDNs Work
• When content is requested from a CDN, the
user is directed to the optimal server
• This is usually done through the DNS, especially for
non-network CDNs
• It can be done though anycasting for network owned
CDNs
• Users who query DNS-based CDNs be returned
different A records for the same hostname
• This is called “mapping”
• The better the mapping, the better the CDN

7. © 2008 Akamai
How CDNs Work
• Example of CDN mapping
• Notice the different A records for different locations:
[NYC]% host www.symantec.com
www.symantec.com CNAME a568.d.akamai.net
a568.d.akamai.net A 207.40.194.46
a568.d.akamai.net A 207.40.194.49
[Boston]% host www.symantec.com
www.symantec.com CNAME a568.d.akamai.net
a568.d.akamai.net A 81.23.243.152
a568.d.akamai.net A 81.23.243.145

8. © 2008 Akamai
End User
Finding the IP Address: The Akamai Way
Akamai High-Level DNS Servers
10 g.akamai.net
1
Browser’
s Cache
OS
2
Local Name
Server
3
example.com’s
nameserver
6
www.example.com
7
a212.g.akamai.net
9
15.15.125.6
16
15
1120.20.123.55
Akamai Low-Level DNS Servers
12 a212.g.akamai.net
30.30.123.5 13
14
4
example.com
.net Root
(InterNIC)10.10.123.55
akamai.net
8

9. © 2008 Akamai
How CDNs Work
• CDNs use multiple criteria to choose the
optimal server
• These include standard network metrics:
• Latency
• Throughput
• Packet loss
• These also include things like CPU load on the
server, HD space, network utilization, etc.
• Geography still counts
• That whole speed-of-light thing
• Should be able to solve that with the next version of
ethernet…

10. © 2008 Akamai
HTML
Object Delivery (Classic CDN)
1. Enduser types www.retailer.com into browser
2. Browser retrieves HTML from retailer’s origin infrastructure
3. HTML instructs browser to get objects from Akamai
4. Browser retrieves images from optimal Akamai EdgeServer
Origin
Infrastructure
End User

11. © 2008 Akamai
HTML
Whole Site Delivery (for Static Sites)
1. Enduser types www.retailer.com into browser
2. Browser retrieves entire site from Akamai cache at the edge
3. HTML instructs browser to get objects from Akamai
4. Browser retrieves images from optimal Akamai EdgeServer
Origin
Infrastructure
End User

12. © 2008 Akamai
HTML
HTML
Whole Site Delivery (for Dynamic Sites)
1. Enduser types www.retailer.com into browser
2. Browser requests HTML from optimal Akamai EdgeServer
3. Akamai EdgeServer retrieves HTML from origin infrastructure
4. Akamai EdgeServer sends HTML to browser
5. Browser retrieves images from optimal Akamai EdgeServer
Origin
Infrastructure
End User

13. © 2008 Akamai
Origin
Infrastructure
End User
HTML
With Akamai:
“Just-in-time caching”
of dynamic and “cold” content by Akamai Pre-fetching
1) Enduser types www.retailer.com into browser
2) Browser requests HTML from optimal Akamai EdgeServer
3) Akamai EdgeServer “GETs” HTML from origin infrastructure
4) Akamai EdgeServer parses HTML and requests uncached images from origin
while delivering HTML to browser
5) EdgeServer delivers “Akamaized” images to browser

14. © 2008 Akamai
Why CDNs Peer with ISPs
• The first and foremost reason to peer is
improved performance
• Since a CDN tries to serve content as “close” to the
end user as possible, peering directly with networks
(over non-congested links) obviously helps
• Peering gives better throughput
• Removing intermediate AS hops seems to give
higher peak traffic for same demand profile
• Might be due to lower latency opening TCP windows
faster
• Might be due to lower packet loss

15. © 2008 Akamai
Why CDNs Peer with ISPs
• Redundancy
• Having more possible vectors to deliver content
increases reliability
• Burstability
• During large events, having direct connectivity to
multiple networks allows for higher burstability than
a single connection to a transit provider
• Burstability is important to CDNs
• One of the reasons customers use CDNs is for
burstability

16. © 2008 Akamai
Why CDNs Peer with ISPs
• Peering reduces costs
• Reduces transit bill (duh)
• Network Intelligence
• Receiving BGP directly from multiple ASes helps
CDNs map the Internet
• Backup for on-net servers
• If there are servers on-net, the IX can act as a
backup during downtime and overflow
• Allows serving different content types

17. © 2008 Akamai
Why ISPs peer with CDNs
• Performance
• CDNs and ISPs are in the same business, just on
different sides - we both want to serve end users as
quickly and reliably as possible
• You know more about your network than any CDN
ever will, so working with the CDN directly can help
them deliver the content more quickly and reliably
• Cost Reduction
• Transit savings
• Possible backbone savings

18. © 2008 Akamai
How Non-Network CDNs use IXes
CDN Servers Transit
Peer Network
• Non-network CDNs do
not have a backbone, so
each IX instance is
independent
• The CDN uses transit to
pull content into the
servers
• Content is then served
to peers over the IX
Origin Server
IX
Content

19. © 2008 Akamai
How CDNs use IXes
• Non-network CDNs usually do not announce
large blocks of address space because no one
location has a large number of servers
• It is not uncommon to see a single /24 from a CDN
at an IX
• This does not mean you will not see a lot of
traffic
• How many web servers does it take to fill a gigabit
these days?

20. © 2008 Akamai
Questions?
• ck@akamai.com